<html>
<head><meta charset="utf-8"><title>Ryan and Mark meeting · t-compiler/performance · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/index.html">t-compiler/performance</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Ryan.20and.20Mark.20meeting.html">Ryan and Mark meeting</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="244708884"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Ryan%20and%20Mark%20meeting/near/244708884" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Ryan.20and.20Mark.20meeting.html#244708884">(Jul 02 2021 at 14:19)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> and I had a great meeting today with the general goal of me taking on a bit more responsibility in the running and further development of perf.rlo and perf related processes. Here are some of the outcomes of our discussions for everyone's awareness:</p>
<ul>
<li><strong>Highest priority work:</strong> better data representation and automatic analysis. There's general consensus that it's not always clear how perf data should be interpreted. We want to make it easy for anyone who opens a PR to see how their PR is <em>actually</em> affecting performance. Perhaps first on this list will be quantifying noise on a per benchmark basis. </li>
<li><strong>Second highest priority</strong>: continuing to improve the perf bot and further improving the perf process around perf runs done on PRs. This may see work before data representation work since it's still high priority, and there is low hanging fruit. </li>
<li><strong>Other work</strong>: this is work we hope to be able to do if/when we have time: <ul>
<li>helping run the perf infrastructure locally both for local development of perf.rlo but also (perhaps more importantly) just for anyone wanting to benchmark their local compilers on their local machine </li>
<li>collecting more stats (though not necessarily displaying them yet). Before we display more stats we want to make progress on better data representation. For example, <span class="user-mention" data-user-id="124287">@mw</span> and I have a small side project for collecting more stats on incr-compilation overhead.</li>
<li>Investigating sandboxing so that eventually we don't need a human to kick off perf runs. Maybe they can just be a part of CI....</li>
</ul>
</li>
<li><strong>Other work along the way</strong>: These are things we hope to be able to do in service of achieving the goals above:<ul>
<li>Improved performance of the service</li>
<li>Easier manipulation of displayed data (i.e., allow different views on the same data)</li>
<li>More easily allow running locally with synthetic or historical data to make local development of perf.rlo easier.</li>
</ul>
</li>
</ul>



<a name="244712893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Ryan%20and%20Mark%20meeting/near/244712893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Ryan.20and.20Mark.20meeting.html#244712893">(Jul 02 2021 at 14:46)</a>:</h4>
<blockquote>
<p>Perhaps first on this list will be quantifying noise on a per benchmark basis. </p>
</blockquote>
<p>If we're willing to have false negatives for significant changes within the noise range (such as a change that shifts the mean but only within a standard deviation of the noise) then that shouldn't be too complex. Otherwise it would take multiple samples to detect such a shift after it occurred which makes it more difficult to attribute it to a specific commit.</p>
<blockquote>
<p>collecting more stats</p>
</blockquote>
<p>Is measuring runtime performance of generated code as opposed to measuring compiler performance in or out of scope for perf rlo?</p>



<a name="244713288"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Ryan%20and%20Mark%20meeting/near/244713288" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Ryan.20and.20Mark.20meeting.html#244713288">(Jul 02 2021 at 14:49)</a>:</h4>
<p>We did discuss runtime benchmarking, and while this is absolutely something we _have_ to do in the long run, it's too much work at the current moment to prioritize.</p>



<a name="244718275"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Ryan%20and%20Mark%20meeting/near/244718275" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Ryan.20and.20Mark.20meeting.html#244718275">(Jul 02 2021 at 15:24)</a>:</h4>
<p>Oh I'd love to see ownership of perf.rlo get spread out a bit more. <span class="user-mention" data-user-id="116122">@simulacrum</span> has way too much on their plate</p>



<a name="244718282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Ryan%20and%20Mark%20meeting/near/244718282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Ryan.20and.20Mark.20meeting.html#244718282">(Jul 02 2021 at 15:24)</a>:</h4>
<p>this is super exciting to me</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>